Spatial construction using guided surface detection

ABSTRACT

Described herein are a system and methods for efficiently using depth and image information for a space to generate a 3D representation of that space. In some embodiments, an indication of one or more points is received with respect to image information, which is then mapped to corresponding points within depth information. A boundary may then be calculated to be associated with each of the points based on the depth information at, and surrounding, each point. Each of the boundaries are extended outward until junctions are identified as bounding the boundaries in a direction. The system may determine whether the process is complete or not based on whether any of the calculated boundaries are currently unlimited in extent in any direction. Once the system determines that each of the boundaries is limited in extent, a 3D representation of the space may be generated based on the identified junctions and/or boundaries.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/866,801, filed May 5, 2020, which is a continuation of U.S. patentapplication Ser. No. 15/988,891, filed May 24, 2018, which has beengranted as U.S. Pat. No. 10,679,372, issued Jun. 9, 2020, the entirecontents of which are hereby incorporated by reference in its entiretyfor all purposes.

BACKGROUND

Three-dimensional (3D) models (e.g., 3D representations of buildingspaces) are often used in a number of architectural and engineeringapplications. As 3D models for a particular space are often notavailable, 3D models must be newly generated for each of these spaces.In some cases, this involves the use of a drafter, who models the spaceby manually using a computer aided drafting (CAD) application. A numberof automated systems are also available that use laser scanners or othersensors for acquisition of 3D data. However, these systems often collectpoint-cloud data which includes an unnecessarily large number of datapoints, making these systems memory intensive and inefficient.

Systems for generating 3D models of indoor spaces face a number ofadditional technical challenges. For example, these systems are oftenunable to distinguish the space from objects within that space. In somecases, users of a system may be forced to remove objects from the spacebefore modeling the space to obtain an accurate model. Some systems maybe capable of automatically extrapolating out point cloud data toestimate the bounds of a space. However, these systems often justidentify the most likely candidate for each structural feature (e.g.,walls, floors, and ceilings) of the space and generate a 3D model of thespace from those likely structural features. This often results in thesystem disregarding atypical structural features of a space as“clutter,” and results in generation of a 3D model that lacks thoseatypical structural features. As a result, these systems are usuallyonly able to generate 3D models of conventional spaces, making thesystems unusable for a number of spaces.

Embodiments of the invention address these and other problems,individually and collectively.

BRIEF SUMMARY

Techniques described herein are directed to a system and methods forefficiently using depth information for a space to generate a 3Drepresentation of that space. In particular, embodiments may involveobtaining both image information as well as depth information for thespace. An indication of one or more points is received with respect tothe image information, which is then mapped to corresponding pointswithin depth information. The described system then calculates aboundary to be associated with each of those one or more points based onthe depth information at, and surrounding, each point. Each of theboundaries are extended outward until junctions (e.g., an intersectionof two boundaries) are identified as limiting the extent of theboundaries in one direction. The system may determine whether theprocess is complete or not based on whether any of the calculatedboundaries are currently unlimited in any direction (e.g., stretch foran infinite distance in any direction). Once the system determines thatthe process is complete, a 3D representation of the space may begenerated based on the identified junctions and/or boundaries.

One embodiment of the disclosure is directed to a method of generating a3D representation of a space comprising receiving an indication of anumber of points, each of the points corresponding to a location upon asurface of a structural feature within the space, determining, for eachof the number of points, a number of corresponding boundaries that matchthe surface of the corresponding structural feature for at least someamount of area, identifying, from the determined number of correspondingboundaries, multiple pairs of intersecting boundaries, generating a setof junctions, wherein each junction is generated as an intersection of apair of intersecting boundaries of the multiple pairs of intersectingboundaries, and after determining that each of the number ofcorresponding boundaries is completely limited in its extent byjunctions within the set of junctions, generating the 3D representationof the space using the set of junctions.

Another embodiment of the disclosure is directed to a system comprisingone or more camera devices, a processor, and a memory includinginstructions. In this system, the instructions, when executed with theprocessor, may cause the system to obtain, from the one or more cameradevices, a depth information associated with a scene, receive anindication of a point within the depth information, calculate, using thedepth information, a first boundary associated with the indicated point,determine one or more bounds for the first boundary based on at leastone second boundary obtained in relation to the scene, and generate a 3Drepresentation of the scene based at least in part on the one or morebounds.

Yet another embodiment of the disclosure is directed to an apparatuscomprising a camera device configured to capture image information, adepth sensor device configured to capture depth information, a mobileapplication stored in a computer-readable medium. The mobileapplication, when executed, may cause the apparatus to receive depthinformation from the depth sensor which corresponds to image informationcaptured using the camera device, receive an indication, via the imageinformation, of a first point and a second point within the depthinformation, identify, using the depth information, a first boundaryassociated with the first point and a second boundary associated withthe second point within the depth information, determine a junction as aline on which the first boundary and second boundary intersect, andcause a 3D model to be generated that includes at least the determinedjunction.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 depicts an illustrative overview of an example system in whichguided surface detection may be used to model a cluttered 3D space inaccordance with at least some embodiments;

FIG. 2 depicts a system architecture for a system that may beimplemented to perform the functionality described in accordance with atleast some embodiments;

FIG. 3 depicts a flow chart that illustrates an example process forgenerating a 3D representation of a space that may be implemented inaccordance with at least some embodiments;

FIG. 4 depicts a technique for calculating boundary data using depthinformation received via a sensor on a user device in accordance with atleast some embodiments;

FIG. 5 depicts a technique for calculating a junction for two surfaceswith respect to a common origin point in accordance with at least someembodiments;

FIG. 6 depicts a technique for bounding surfaces using calculatedjunction information in accordance with at least some embodiments;

FIG. 7 depicts an illustrative example of an interaction that may occurusing the system described herein in accordance with at least someembodiments;

FIG. 8 depicts a flow diagram which illustrates an example process forgenerating a 3D representation of a space using depth and imageinformation obtained with respect to the space in accordance with atleast some embodiments; and

FIG. 9 depicts an illustrative example of a user device capable ofperforming at least a portion of the functionality described herein.

DETAILED DESCRIPTION

In the following description, various embodiments will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described.

FIG. 1 depicts an illustrative overview of an example system in whichguided surface detection may be used to model a cluttered 3D space inaccordance with at least some embodiments. In FIG. 1 , a user device 102is depicted as being operated within a space 104. The space 104 isdepicted as including a number of obstacles 106 (e.g., clutter), whichmay block a view of one or more structural features (e.g., a wall,floor, or ceiling) of the space 104. The user device 102 may be incommunication with a mobile application server 108.

For clarity, a certain number of components are shown in FIG. 1 . It isunderstood, however, that embodiments of the invention may include morethan one of each component. In addition, some embodiments of theinvention may include fewer than or greater than all of the componentsshown in FIG. 1 . In addition, the components in FIG. 1 may communicatevia any suitable communication medium (including the internet), usingany suitable communication protocol.

In some embodiments, the user device 102 may include a mobileapplication that, when executed, causes the user device 102 to captureinput sensor data from a number of input sensors in relation to thespace 104. By way of example, the user device 102 may capture imageinformation, depth information, and geographic location information(e.g., GPS coordinates) with respect to the space 104 and/or the userdevice 102. Additionally, the user device 102 may be configured todisplay at least a portion of the obtained input information to a user.For example, a display on the user device 102 may be used to presentimage information captured by a camera installed on the user device 102.The image information may be obtained by the user device 102 in parallelto, and to correspond with, depth sensor output (e.g., a depth map)obtained using a depth sensor installed on the user device 102.

In some embodiments, the mobile application server 108 may be configuredto receive the input sensor output from the user device 102 and generatea 3D representation 110 of the space 104. For example, the user device102 may obtain depth information and location information from the userdevice 102. The mobile application server 108 may also receive anindication of at least one point selected by the user within the depthinformation that may be used to identify at least one boundary. In someembodiments, the mobile application server 108 may receive informationrelated to a number of boundaries identified by the user device 102. Themobile application server 108 may then identify lines which representjunctions along which each two boundaries intersect. The 3Drepresentation 110 may then be generated by the mobile applicationserver 108 using this data. For purposes of this disclosure, the term“line” is not limited to a geometrically straight line. The term “line”can encompass a straight line (e.g., at the intersection of two surfacesthat happen to be rectangular and planar) or can encompass curved linesor lines that are not straight in situations where a surface is curvedor round or not entirely straight where the junction between surfacesotherwise is not straight.

In accordance with at least some embodiments, a user of, and/or anaccount associated with, the user device 102 may be identified. Theaccount may be one that is maintained on behalf of the user by themobile application server 108. In some embodiments, the user/account maybe identified based on a phone number or serial number associated withthe user device 102. In some embodiments, the user may be asked to signinto an account upon or after execution of a mobile application on theuser device 102, such that any actions performed using the mobileapplication may be automatically associated with the logged account. Insome embodiments, the identity of the user can be determined andverified more efficiently using biometric information detected by theuser device 102 (e.g., finger-print or thumb-print detection, facialrecognition, iris scan, or the like).

In some embodiments, the 3D representation 110 of the space 104 mayinclude one or more measurement values for the space 104. For example,after identifying a number of junctions along which each two boundariesintersect, the mobile application server 108 may determine distancesbetween each junction based on the provided depth information. In someembodiments, information related to the 3D representation's 110 position(e.g., orientation and location) in space may be determined. Forexample, the mobile application server 108 may receive locationinformation from the user device 102. In this example, a relativelocation of each junction and/or boundary to the user device 102 may bedetermined from the depth sensor output and used (in conjunction withthe user device location) to calculate a position of the 3Drepresentation that corresponds to a position of the actual spacerepresented by the 3D representation. The generated 3D representationmay be stored in a number of ways. For example, in some embodiments, the3D representation may be stored as a wireframe representation (e.g.,only the identified junctions are stored). In some embodiments, the 3Drepresentation may be stored as a series of boundary functions (i.e.,mathematical functions that represent the surfaces identified withrespect to the 3D representation).

By way of illustrating interactions between various components depictedin FIG. 1 , consider a scenario in which a user enters a room (i.e., anexample space 104) having a number of obstacles 106, and in which theuser wishes to generate a 3D representation of the room. In thisexample, the user may execute a mobile application installed on his orher user device (e.g., a tablet or mobile phone). The mobile applicationmay cause the user device to activate both a camera device and a depthsensor installed upon the user device in order to capture input relatedto the room. The image information captured by the camera device may bedisplayed upon a display of the user device. In this illustrativeexample, the user may select a number of points 112, 114, and 116 withinthe image information displayed on the user device and which correspondto actual points within the room. This can be accomplished, for example,by touching images of those points 112, 114, and 116 on a touch-screenof the user device. In some cases, the user device 102 may obtain depthinformation for the number of points 112, 114, and 116 as well as depthinformation for the area immediately surrounding those points. In someembodiments, the user device 102 may determine, from the depthinformation a number of boundaries, each of which is associated with oneof the number of points 112, 114, and 116. In these embodiments, theuser device 102 may transmit an indication of the number of boundaries(e.g., a boundary function and a distance to some point on the boundary)to the mobile application server 108. In some embodiments, the userdevice 102 may transmit the obtained depth information to the mobileapplication server 108, and the mobile application server 108 maycalculate the boundaries to be associated with the room. In someembodiments, the user device 102 and/or the mobile application server108 may determine when a 3D representation is complete by determiningwhether each of the boundaries in the 3D representation is limited inits extent (i.e., does not extend infinitely in any direction). Forexample, in some embodiments the 3D representation may be determined tobe complete after detecting that each boundary calculated for the roomis limited in every direction by a junction. In this illustrativeexample, a 3D representation of the room may then be generated.

It should be noted that in the illustrative example above, the userdevice 102 may have installed a motion tracking camera, which tracks therelative position of the image information with respect to the userdevice 102. Accordingly, the user may be able to walk around the roomand reposition the user device 102 while still accurately determining arelative position for each of the indicated boundaries. Additionally,because each boundary is calculated from some point indicated by theuser (e.g., 112, 114, or 116), an accurate 3D representation can begenerated that does not include obstacles, or other clutter, in theroom. In some cases, this even allows a 3D representation to begenerated for a room that has very little exposed wall space (e.g., aroom in which one or more of the walls is occupied by a large piece offurniture) without the need to move any obstacles.

FIG. 2 depicts a system architecture for a system that may beimplemented to perform the functionality described in accordance with atleast some embodiments. In FIG. 2 , a user device 202 may be incommunication with a number of other components, including at least amobile application server 204. The mobile application server 204 mayperform at least a portion of the processing functions required by amobile application installed upon the user device 202. The user device202 may be an example of the user device 102 described in FIG. 1 . Themobile application server 204 may be an example mobile applicationserver 108 described with respect to FIG. 1 .

A user device 202 may be any suitable electronic device that is capableof providing at least a portion of the capabilities described herein. Inparticular, the user device 202 may be any electronic device capable ofidentifying location information with respect to an indicated point. Insome embodiments, a user device may be capable of establishing acommunication session with another electronic device (e.g., mobileapplication server 204) and transmitting/receiving data from thatelectronic device. A user device may include the ability to downloadand/or execute mobile applications. User devices may include mobilecommunication devices as well as personal computers and thin-clientdevices. In some embodiments, a user device may comprise any portableelectronic device that has a primary function related to communication.For example, a user device may be a smart phone, a personal dataassistant (PDA), or any other suitable handheld device. The user devicecan be implemented as a self-contained unit with various components(e.g., input sensors, one or more processors, memory, etc.) integratedinto the user device. Reference in this disclosure to an “output” of acomponent or an “output” of a sensor does not necessarily imply that theoutput is transmitted outside of the user device. Outputs of variouscomponents might remain inside a self-contained unit that defines a userdevice.

In one illustrative configuration, the user device 202 may include atleast one memory 206 and one or more processing units (or processor(s))208. The processor(s) 208 may be implemented as appropriate in hardware,computer-executable instructions, firmware or combinations thereof.Computer-executable instruction or firmware implementations of theprocessor(s) 208 may include computer-executable or machine executableinstructions written in any suitable programming language to perform thevarious functions described. The user device 202 may also include one ormore input sensors 210 for receiving user and/or environmental input.There may be a variety of input sensors 210 capable of detecting user orenvironmental input, such as an accelerometer, a camera device, a depthsensor, a microphone, a global positioning system (e.g., GPS) receiver,etc. The one or more input sensors 210 may include at least a rangecamera (e.g., a depth sensor) capable of generating a range image, aswell as a camera device configured to capture image information. Thecamera device may be a motion tracking camera, which is capable ofmaintaining location information with respect to the captured images.

For the purposes of this disclosure, a depth sensor (e.g., a rangecamera) may be any device configured to identify a distance or range ofan object or objects from the depth sensor. In some embodiments, thedepth sensor may generate a depth image (or range map), in which pixelvalues correspond to the detected distance for that pixel. The pixelvalues can be obtained directly in physical units (e.g., meters). In atleast some embodiments of the disclosure, the 3D imaging system mayemploy a range camera that operates using structured light. In a depthsensor that operates using structured light, a projector projects lightonto an object or objects in a structured pattern. The light may be of arange that is outside of the visible range (e.g., infrared orultraviolet). The depth sensor may be equipped with one or more cameradevices configured to obtain an image of the object with the reflectedpattern. Distance information may then be generated based on distortionsin the detected pattern. It should be noted that although thisdisclosure focuses on the use of a depth sensor using structured light,any suitable type of depth sensor, including those that operate usingstereo triangulation, sheet of light triangulation, time-of-flight,interferometry, coded aperture, or any other suitable technique forrange detection, would be useable by the described system.

The memory 206 may store program instructions that are loadable andexecutable on the processor(s) 208, as well as data generated during theexecution of these programs. Depending on the configuration and type ofuser device 202, the memory 206 may be volatile (such as random accessmemory (RAM)) and/or non-volatile (such as read-only memory (ROM), flashmemory, etc.). The user device 202 may also include additional storage212, such as either removable storage or non-removable storageincluding, but not limited to, magnetic storage, optical disks, and/ortape storage. The disk drives and their associated computer-readablemedia may provide non-volatile storage of computer-readableinstructions, data structures, program modules, and other data for thecomputing devices. In some implementations, the memory 206 may includemultiple different types of memory, such as static random access memory(SRAM), dynamic random access memory (DRAM) or ROM. As used herein, theterm “modules” may refer to programming modules executed by computingsystems (e.g., processors) that are installed on and/or executed from acomputing device such as the user device 202 or the mobile applicationserver 204. Turning to the contents of the memory 206 in more detail,the memory 206 may include an operating system 214 and one or moreapplication programs or services for implementing the features disclosedherein including at least a mobile application 216. The memory 206 mayalso include application data 218, which provides information to begenerated by and/or consumed by the mobile application 216. In someembodiments, the application data 218 may be stored in a database.

For the purposes of this disclosure, a mobile application 216 may be anyset of computer executable instructions installed upon, and executedfrom, a user device 202. Mobile applications may be installed on a userdevice by a manufacturer of the user device or by another entity. Insome embodiments, the mobile application may cause a user device toestablish a communication session with a mobile application server 204that provides backend support for the mobile application. A mobileapplication server 204 may maintain account information associated witha particular user device and/or user. In some embodiments, a user may berequired to log into a mobile application in order to accessfunctionality provided by the mobile application. In some embodiments,the identity of the user can be determined and verified for purposes oflogging into an account associated with the mobile application andassociated with the user by using biometric information detected by theuser device 202 (e.g., finger-print or thumb-print detection, facialrecognition, iris scan, or the like).

In accordance with at least some embodiments, the mobile application 216may be configured to, in conjunction with the processors 208, obtaindepth information in relation to one or more points indicated by a user.In some embodiments, the mobile application 216 may cause the userdevice 202 to display on the user device's display an image captured bya camera of the mobile device. The user may select some point within theimage (e.g., via a touchscreen) and the mobile application 216 mayidentify depth information that corresponds to the selected point. Forexample, both image information and depth information may be collectedin parallel by the user device 202 via two different input sensors. Thetwo sets of information may be correlated such that pixels in one set ofinformation are associated with corresponding pixels in the other set ofinformation. In this way, when a user selects a point within the imageinformation, the depth information associated with the selected pointmay be determined. In some embodiments, the mobile application 216 maycause the depth information (as well as information associated with aposition of the user device 102) to be transmitted to the mobileapplication server 204 for further processing. In some embodiments, themobile application 216 may identify a boundary to be associated witheach point selected by the user. An indication of each of the identifiedboundaries may then be transmitted to the mobile application server 204.

The user device 202 may also contain communications interface(s) 220that enable the user device 202 to communicate with any other suitableelectronic devices. In some embodiments, the communication interface 220may enable the user device 202 to communicate with other electronicdevices on a network (e.g., on a private network). The user device 202may also include input/output (I/O) device(s) and/or ports 222, such asfor enabling connection with a keyboard, a mouse, a pen, a voice inputdevice, a touch input device, a display, speakers, a printer, etc.

In some embodiments, the user device 202 may communicate with the mobileapplication server 204 via a communication network. The communicationnetwork may include any one or a combination of many different types ofnetworks, such as cable networks, the Internet, wireless networks,cellular networks, and other private and/or public networks. Inaddition, the communication network may comprise multiple differentnetworks. For example, the user device 202 may utilize a wireless localarea network (WLAN) to communicate with a wireless router, which maythen route the communication over a public network (e.g., the Internet)to the mobile application server 204.

The mobile application server 204 may be any computing device orplurality of computing devices configured to perform one or morecalculations on behalf of the mobile application 216 on the user device202. In some embodiments, the mobile application 216 may be in periodiccommunication with the mobile application server 204. For example, themobile application 216 may receive updates, push notifications, or otherinstructions from the mobile application server 204. In someembodiments, the mobile application 216 and mobile application server204 may utilize a proprietary encryption and/or decryption scheme tosecure communications between the two. In some embodiments, the mobileapplication server 204 may be executed by one or more virtual machinesimplemented in a hosted computing environment. The hosted computingenvironment may include one or more rapidly provisioned and releasedcomputing resources, which computing resources may include computing,networking, and/or storage devices. A hosted computing environment mayalso be referred to as a cloud-computing environment.

In one illustrative configuration, the mobile application server 204 mayinclude at least one memory 224 and one or more processing units (orprocessor(s)) 226. The processor(s) 226 may be implemented asappropriate in hardware, computer-executable instructions, firmware orcombinations thereof. Computer-executable instruction or firmwareimplementations of the processor(s) 226 may include computer-executableor machine executable instructions written in any suitable programminglanguage to perform the various functions described.

The memory 224 may store program instructions that are loadable andexecutable on the processor(s) 226, as well as data generated during theexecution of these programs. Depending on the configuration and type ofmobile application server 204, the memory 224 may be volatile (such asrandom access memory (RAM)) and/or non-volatile (such as read-onlymemory (ROM), flash memory, etc.). The mobile application server 204 mayalso include additional storage 228, such as either removable storage ornon-removable storage including, but not limited to, magnetic storage,optical disks, and/or tape storage. The disk drives and their associatedcomputer-readable media may provide non-volatile storage ofcomputer-readable instructions, data structures, program modules, andother data for the computing devices. In some implementations, thememory 224 may include multiple different types of memory, such asstatic random access memory (SRAM), dynamic random access memory (DRAM)or ROM. Turning to the contents of the memory 224 in more detail, thememory 224 may include an operating system 230 and one or moreapplication programs or services for implementing the features disclosedherein including at least a module for identifying one or moreboundaries that make up a space (boundary detection module 232), and amodule for generating a 3D representation of a space (3D modeling module234). The memory 206 may also include server-side databases, such as adatabase of account data 236 and/or a database of 3D representations238.

The memory 224 and the additional storage 228, both removable andnon-removable, are examples of computer-readable storage media. Forexample, computer-readable storage media may include volatile ornon-volatile, removable or non-removable media implemented in any methodor technology for storage of information such as computer-readableinstructions, data structures, program modules or other data. The mobileapplication server 204 may also contain communications connection(s) 240that allow the mobile application server 204 to communicate with astored database, another computing device or server, user terminals,and/or other components of the described system. The mobile applicationserver 204 may also include input/output (I/O) device(s) and/or ports242, such as for enabling connection with a keyboard, a mouse, a pen, avoice input device, a touch input device, a display, speakers, aprinter, etc.

Turning to the contents of the memory 224 in more detail, the memory 224may include a boundary detection module 232, a 3D modeling module 234, adatabase of account data 236, and/or a database of 3D representations238.

In some embodiments, the boundary detection module 232 may be configuredto, in conjunction with the processors 226, receive depth informationand potentially location information from the user device 202 andcalculate a number of appropriate planes from the received data. In someembodiments, the boundary detection module 232 may receive depthinformation related to a point selected by a user as well as depthinformation for an area surrounding that point. Some example techniquesfor calculating a boundary from the received data are described belowwith respect to FIG. 4 . In some embodiments, the 3D modeling module canreceive from a user device 202 an indication of at least one of theidentified boundaries and utilize each such indication to generate a 3Dmodel of the corresponding space.

In some embodiments, the 3D modeling module 234 may be configured to, inconjunction with the processors 226, generate a 3D model of a spaceusing the boundaries calculated by the boundary detection module 232. Insome embodiments, this may involve identifying junctions at whichboundaries intersect. In some cases, the 3D modeling module 234 may beconfigured to continue to receive boundary data until a number ofjunctions are identified so that each of the boundaries is limited inits extent by junctions (e.g., no boundary stretches for an infinite orextremely large length in any direction). In some embodiments, the 3Dmodeling module 234 may be a computer aided drafting application whichhas been configured to perform at least a portion of the techniquesdescribed herein.

FIG. 3 depicts a flow chart that illustrates an example process forgenerating a 3D representation of a space that may be implemented inaccordance with at least some embodiments. Some or all of the process300 (or any other processes described herein, or variations, and/orcombinations thereof) may be performed under the control of one or morecomputer systems configured with executable instructions and may beimplemented as code (e.g., executable instructions, one or more computerprograms, or one or more applications) executing collectively on one ormore processors, by hardware or combinations thereof. The code may bestored on a computer-readable storage medium, for example, in the formof a computer program comprising a plurality of instructions executableby one or more processors. The computer-readable storage medium may benon-transitory. Process 300 may be performed by an example user device202, a mobile application server 204, and various other components, eachof which is depicted with respect to FIG. 2 , or can be performed usingdistributed processing techniques so that a combination of a user device202, a mobile application server 204 and/or various other components,each of which performs only part of the overall process, cooperate toperform the overall process 300.

Process 300 may begin at 302, when the system receives sensor input, aswell as position data, obtained from a user device. As describedelsewhere, the sensor input may include image information as well asdepth sensor output. In some embodiments, the sensor input may bereceived as a stream of data. For example, the input sensor data may bereceived as a video stream. In some embodiments, at least a portion ofthe process 300 described herein may be performed at a user device. Forexample, a user device may receive in parallel, via a depth sensor and acamera, both depth information and image information for a scene. Inthis example, the depth information and the image information may beassociated, in that pixels within the depth information correspond topixels within the depth information. The user device may further displaythe image information on its display.

At 304, input may be received which indicates a particular point. Insome embodiments, the input is received via an indication on the display(e.g., a user's touch on a touchscreen device or a cursor selection of apoint) with respect to image information. Based on this indication, apoint within the depth information may be identified that corresponds tothe point indicated with respect to the image information. For example,a pixel or pixels that are located in a position within the depthinformation that corresponds to the position of the indicated pointwithin the image information may be determined. These pixels within thedepth information may be assigned a value that corresponds to a depth ordistance of the point from the user device.

At 306, a boundary may be calculated for the indicated point. An exampleof a technique that may be used to calculate a boundary for a givenpoint within depth information is described below with respect to FIG. 4. The boundary may be identified and stored in memory as an equation orfunction. In some embodiments, a common point of reference (e.g., anorigin point) may be identified such that any subsequently calculatedboundary is calculated with respect to the common point of reference. Insome embodiments, the common point of reference may be set to a positionin space at which the user device was located at the time that theprocess 300 was initiated. Each boundary may be identified by thedirections in which the boundary extends and an offset from the commonpoint of reference.

At 308, it may be determined whether the calculated boundary intersectswith one or more other boundaries calculated in a similar manner. If theboundary is determined not to intersect with another boundary (e.g., theone or more other boundaries are all parallel to the boundary) then theuser device continues to obtain sensor input. If the boundary isdetermined to intersect with one or more other boundaries, thenjunctions may be determined for each intersecting boundary at 310. Anexample of a technique that may be used to determine whether a junctionfor two boundaries is described in greater detail below with respect toFIG. 5 .

At 312, it may be determined whether the set of junctions is complete bydetermining whether or not all boundaries have been fully limited sothey don't extend infinitely in any direction. A boundary may be limitedby a junction in that any portion of the boundary which is separatedfrom the indicated point on the boundary by a junction may be discardedor removed. This may be repeated for the boundary using a number ofjunctions. A boundary may be determined to be fully limited if theboundary has a finite length in each direction as defined by a junction.In some embodiments, a 3D representation of a space may be generated at314 from the set of junctions and/or boundaries after determining thatthe set of junctions is complete.

FIG. 4 depicts a technique for calculating boundary data using depthinformation received via a sensor on a user device in accordance with atleast some embodiments. For the purposes of this disclosure, a boundarymay be any edge or border that bounds some portion of a 3D space. Insome cases, a boundary may be defined by a plane or other flat surface.The techniques described with respect to FIG. 4 are directed toidentifying a boundary that is defined by a plane. However, one skilledin the art would recognize that there are a number of ways ofidentifying boundaries that align with surfaces of structures, some ofwhich may be applied to surfaces that are not planar or flat.

In FIG. 4 , a boundary origin point P may be selected by a user. In someembodiments, the boundary origin point may be selected by the user on auser device by selecting a point corresponding to that boundary originpoint within image information displayed on the user device. Afterselection of the boundary origin point P, the user device may, using adepth sensor, collect depth information for points immediatelysurrounding the boundary origin point P. For example, the user devicemay collect depth information for points Q and R so that points Q and Rform a 90° angle with respect to the boundary origin point P (though a90° angle is not required to determine a boundary). The points Q and Rmay be selected such that some distance d₁ between point Q and theboundary origin point P is equal to the distance d₂ between point R andthe boundary origin point P (e.g., such that each of d₁ and d₂represents one unit of length). Additionally, D_(o) is a depth (i.e., adistance) associated with the boundary origin point P, D₁ is a depthassociated with point Q, and D₂ is a depth associated with point R. Theuser device may calculate coordinates for each of the points withrespect to space (e.g., coordinates along an X, Y, and Z axis). In someembodiments, the coordinates for the three points may be determined withrespect to some point of origin common to each of the boundaries to begenerated. For example, the coordinates for the three points may bedetermined with respect to a position of a user device upon initiationof the techniques described. To determine coordinates of the threepoints, the user device may use orientation data (e.g., obtained from acompass of the user device) and depth information to determine adistance and direction for each of the points to the user device.

By way of illustrating one embodiment of the techniques describedherein, the system may select three points in space as [X₀, Y₀, Z₀],[X₁, Y₁, Z₁], and [X₂, Y₂, Z₂]. In this example, the system may identifyvectors associated with the points and then find a cross product betweenthose vectors. By way of illustration, vector

may be identified as [X₁−X₀, Y₁−Y₀, Z₁−Z₀,] and vector

may be identified as [X₂−X₀, Y₂−Y₀, Z₂−Z₀,]. A cross product may then becalculated for vectors

and

in order to determine a normal vector (e.g., a vector which isperpendicular to a boundary that includes the vectors

and

). The normal vector may then be used along with one of the points tocalculate an equation for the boundary. For example, if the normalvector is [X_(N), Y_(N), Z_(N)], then the equation for the boundary (aplanar boundary in this example) is:

X _(N)(X−X ₀)+Y _(N)(Y−Y ₀)+Z _(N)(Z−Z ₀)=0

It should be noted that a number of techniques can be used to identify aboundary that includes a boundary origin point P using depth sensor andorientation information from a user device.

FIG. 5 depicts a technique for calculating a junction for two planarboundaries with respect to a common origin point in accordance with atleast some embodiments. In FIG. 5 , the system may identify two or moreboundaries relevant to a space. In some embodiments, the system maydetermine functions that each represent a boundary with respect to twodifferent boundary origin points as described with respect to FIG. 4above. For example, a user may select two boundary origin points 502 and504. The system may then generate respective planar boundaries 506 and508, or functions which represent the boundaries, in accordance with thetechniques described above. The generated boundaries 506 and 508 may beboundless, in that each of the boundaries may continue for an infinitelength in each direction.

A junction 510 that bounds part of the two boundaries 502 and 504 is aline that lies within both boundaries. To bound the boundaries using thejunction, the portions of the boundaries 506 and 508 which are separatedfrom the two boundary origin points 502 and 504 by the junction arediscarded or removed. In some embodiments, the junction 510 may becalculated by setting the equations representing each of the respectiveboundaries equal to each other. It should be noted that a junction mayonly be identified for two boundaries that are not parallel (i.e., thetwo boundaries must intersect at some point). It should also be notedthat if the junction of the boundaries is a large distance away from theorigin (e.g., greater than some threshold distance), then the boundariesmay be considered parallel for the purposes of this disclosure. In thiscase, the system may determine that additional junctions are neededbefore the boundaries can be considered limited in their extent.

In some embodiments, the system may maintain an indication of a point oforigin 512 that is common to each of the boundaries. In someembodiments, the point of origin 512 may be determined as a point inspace of a user device that initiated the techniques described herein.In some embodiments, the user device may use a motion tracking camera(e.g., a camera that includes accelerometers and a compass) to keeptrack of the point of origin 512 with respect to a current position(e.g., orientation and location) of the user device. Each boundaryselected by a user may be identified with respect to the point of origin512. In this way, a user is able to move the user device around withoutlosing the relative position of each identified boundary.

FIG. 6 depicts a technique for generating a model of a space by boundingboundaries using calculated junction information in accordance with atleast some embodiments. In FIG. 6 , a space is depicted as having anatypical structure. An atypical structure is a structure that variesfrom a conventional structure in that it includes some anomaly (e.g., aprotrusion or depression) that does not commonly occur. In the exampledepicted in FIG. 6 , a protrusion is depicted as being within a cornerof the space. To generate a 3D representation of this corner, a user mayinitiate a mobile application on his or her user device at point 602.The user may then view image information related to the space to bemapped on a user device via a graphical user interface (GUI) of themobile application. Although the space is depicted as two dimensional(in a manner similar to a partial floor-plan), it should be noted thatthe space represents a three dimensional area. The user may then selectimage information that corresponds to each of points 604, 606, 608, and610. Depth sensor output, as well as positional information associatedwith the user device, may then be used to identify actual points 604,606, 608, and 610 within the space. It should be noted that although theinformation generated with respect to this process may be generated inrelation to point 602, the user may be free to move around the space andselect points from different angles.

After all of points 604, 606, 608, and 610 have been selected by theuser, boundaries corresponding to each of those points may beidentified. Example techniques for identifying a boundary with respectto a selected point (e.g., a boundary origin point) are describedelsewhere in this disclosure. Once a number of boundaries have beengenerated with respect to points 604, 606, 608, and 610, junctions maybe determined with respect to each intersection of two boundaries.Example techniques for generating junctions with which to boundboundaries are also described elsewhere in this disclosure.

In some embodiments, a 3D representation of a space may be generated bybounding the space using a number of boundaries and junctions. To dothis, the user may need to select a boundary for every surface of astructural feature within the room (e.g., walls, ceiling, floor, beam,etc.). In some embodiments, the 3D representation may be consideredincomplete if one or more boundaries are left with their extentunlimited in any direction. In some embodiments, as a user selects a newpoint, the system generates a boundary associated with that point andextends that boundary in all directions either until it intersectsanother boundary (e.g., at a junction) or ad infinitum (or up to amaximum distance). Where the boundary meets another boundary, both ofthe intersecting boundaries are limited in extent such that the portionof the boundary which is separated from the selected point (e.g., points604, 606, 608, and 610) is removed. In some embodiments, a boundary maybe limited in its extent more than once. For example, a user may selectpoints 604 and 610 prior to selecting points 606 and 608. In thisexample, the boundaries corresponding to points 604 and 610 may first belimited based on the intersection of those two boundaries. Once the userhas subsequently selected points 606 and 608, the boundaries may belimited again such that the portion of the boundaries (e.g., 618) thatlie between the new junction and the previous junction may be discarded.In this way, a 3D representation may be updated to include an indicationof an atypical structural feature.

Using the techniques described herein, a user may map any space composedof any combination of boundaries. It should be noted that the FIG. 6depicts an atypical structure in that the space includes multipleparallel boundaries which are offset. Such an atypical structure islikely to be mistaken as clutter by a conventional space modelingsystem. Hence, embodiments of the current system are advantageous oversuch conventional systems.

FIG. 7 depicts an illustrative example of an interaction that may occurusing the system described herein in accordance with at least someembodiments. In FIG. 7 , a user device 702 is depicted as being used tocapture image information related to a scene 704. In FIG. 7 , the scene704 for which image information has been captured includes a number ofobstacles 706 that block a view of at least a portion of somestructures. In some embodiments, the functionality described withrespect to FIG. 7 may be enabled via a mobile application installed uponthe user device 702.

As depicted in FIG. 7 , a user is able to select several points 708,710, and 712 which correspond to unobstructed portions of a surface of astructural feature. The system may be configured to generate, afterreceiving an indication of the points 708, 710 and 712, correspondingboundaries that match the surface of the corresponding structuralfeature for at least some amount of area. This is described in greaterdetail with respect to FIG. 4 above. When these boundaries are extendedoutward, the system may determine one or more junctions at which thegenerated boundaries intersect. For example, if the boundarycorresponding to point 712 and the boundary corresponding to point 708are each extended until they intersect, the system would approximate thecorner 716 of the room without needing to receive an indication of thatcorner from the user and despite any obstacles that may be obstructing aview of the corner. Similarly, if the boundary corresponding to point712 and the boundary corresponding to point 710 are each extended untilthey intersect, the system would approximate the corner 718 of the room.Information determined for each of these corners 716 and 718 may bestored as junctions.

In some embodiments, the mobile application installed upon the userdevice 102 may enable a user to mark locations of various structuralfeatures. For example, the user may mark the location of a window 720 ora door 722. In this example, the user may mark each of the corners ofthe respective window 720 or door 722 in order to identify the bounds ofthe feature. The system may then add the indicated structural featuresto any subsequently generated 3D representation of the space 704. Insome embodiments, the system may also store an indication of one or moreobstacles 706 and its respective location within the space 704.

FIG. 8 depicts a flow diagram which illustrates an example process forgenerating a 3D representation of a space using depth and imageinformation obtained with respect to the space in accordance with atleast some embodiments. Process 800 may be performed using somecombination of a user device 202 and/or a mobile application server 204,each of which is depicted with respect to FIG. 2 .

Process 800 may begin at 802 when sensor output is received. Inaccordance with at least some embodiments, the sensor output may includeat least image information as well as depth information. The sensoroutput may be obtained using multiple sensor devices installed on asingle user device. In some embodiments, the sensor output may bereceived as streaming data (e.g., data that is constantly updated). Insome embodiments, the sensor output may be received as a single stillimage.

At 804, the process may involve receiving indications of points withinthe obtained sensor output. In some embodiments, the indication can bereceived via a touch on a display of the user device at a point withinimage information. Based on this indication, a point within the depthinformation may be identified that corresponds to the point indicatedwith respect to the image information. For example, a determination canbe made that pixel or pixels that are located in a position within thedepth information correspond to the position of the indicated pointwithin the image information.

At 806, the process may involve calculating boundaries (planarboundaries in this example) for each of the indicated points. An exampleof a technique that may be used to calculate a boundary for a givenpoint within depth information is described above with respect to FIG. 4. In some embodiments, each boundary may be represented by an equationor function. It should be recognized that one skilled in the art wouldbe aware of a number of techniques for finding a line or junction thatexists at the intersection of two intersecting boundaries.

At 808, the process may involve identifying multiple pairs ofintersecting boundaries within the calculated boundaries. An example ofa technique that may be used to determine whether a junction for twoboundaries is described in greater detail above with respect to FIG. 5 .The process 800 may involve identifying one junction for eachintersecting pair of boundaries. It should be noted that a junction maybe removed from the set of junctions after determining that it isoutside of the bounds of the space (e.g., another junction has causedthe junction to no longer bound any boundaries).

At 810, the process may involve generating a set of junctions based onintersections between each pair of intersecting boundaries. At 812, theprocess may involve generating a 3D representation from the junctions inthe set of junctions, the suitably limited boundaries, or somecombination of the two. In some embodiments, this step may be performedused a computer aided drafting (CAD) application.

FIG. 9 depicts an illustrative example of a user device capable ofperforming at least a portion of the functionality described herein. InFIG. 9 , a front 902(A) and back 902(B) is depicted for a user device902. The depicted user device 902, as may be used in some particularembodiments of the system described herein, may be a ZENFONE AR(ZS571KL) smartphone device manufactured by ASUS corporation or a PHAB 2PRO smartphone device manufactured by LENOVO corporation.

As depicted in FIG. 9 , the user device 902 may include a display screen904 capable of displaying image information to a user of the user device902. Additionally, the user device 902 may include a number of cameradevices. For example, the user device 902 may include a front-facingcamera 906. Additionally, the user device 902 may include multiplerear-facing cameras, each of which serves different purposes. Forexample, the rear-facing cameras of the user device 902 may include botha high-resolution camera device 908 for capturing detailed images, amotion tracking camera 910 for tracking the user device's location as itmoves through space while capturing image information, and a depthsensor camera 912 for capturing depth information associated with acaptured image information.

Although the foregoing examples demonstrate use of the foregoing systemsand processes on planar boundaries, the foregoing systems and processesare not limited to uses on flat, planar boundaries. Those systems andprocesses can be adapted for use on non-planar boundaries that meet withother boundaries at one or more junctions. For curved surfaces, forexample, the systems and processes can receive an indication from a userof a point on a curved boundary (e.g., based on a touch at thecorresponding point on the display screen where an image of the curvedsurface appears) and an indication (through menu options and/or a touchor click, or otherwise) that the boundary is curved. The system andprocess can prompt the user to select at least three more points atdifferent locations on the same boundary using any one or more of thepreviously described systems and techniques. The systems and processesthen can calculate (e.g., through suitable programming and processing) aBezier curve that represents the boundary's curvature and generate(based on junctions and/or other surfaces determined using the foregoingsystems and processes and the Bezier curve) a 3D representation of thecurved surface.

In some embodiments, a boundary may be defined by a plane (e.g., aninfinitely large flat surface). In some embodiments, the boundary may bedefined by a non-flat surface. For example, the boundary may be definedby a curve created using some function. By way of illustration, aboundary may be defined by a Bezier curve. In at least some of theseembodiments, a function may be created by obtaining points along asurface and fitting a function to those points. This may involveobtaining multiple points along the same surface to act as controlpoints for a curve. For example, a user may select a number of points ona curved surface. In this example, a relative depth of each of thepoints and/or a curvature at each point may be determined using depthsensor data in the manners described above. The positions of theselected points may then be used to create a polynomial (e.g., aBernstein polynomial) that defines a curve.

In some embodiments, multiple curves may be defined for a single surfacethat are used to create a composite curve. A boundary associated withthe surface may then be defined by the composite curve. For example, acurvature of a surface may be determined for each of multiple pointsalong a surface. In this example, a low order polynomial may be fittedto each of the points along the surface. The low order polynomials maythen be combined to create a composite curve (e.g., a “path”).

By way of illustration, consider a scenario in which a user wishes todefine a boundary for a curved wall within a room. Assume, for thisscenario, that the wall curves around a vertical axis. In this scenario,the user may, instead of selecting a single point on that surface,select four points along the surface. The four points may be at the samerelative height on the wall or at different heights. In this example,depth information collected with respect to each of the selected pointsmay be used to fit a Bezier curve to the surface of the wall. A boundarymay then be determined as being defined by the Bezier curve.

Additionally, the user device 902 may include software that, inconjunction with a number of processors of the user device 902, providesat least a portion of the functionality described herein. For example,the software application TANGO, which is developed by GOOGLEcorporation, enables motion tracking, area learning, and depthperception functionality on the depicted user device 902. A mobileapplication, as described herein, which is installed upon the userdevice 902 may use one or more of these functionalities by performing anAPI or method call in accordance with TANGO specifications. Accordingly,it should be noted that the system described herein is fully enabled bythe combination of hardware and software depicted.

Embodiments of the invention provide for a number of technicaladvantages over conventional systems. Conventional systems whichgenerate 3D representations of a space often use devices (e.g., LIDAR)that scan an entire space as well as all of the items within that space.This results in the collection of a large amount of point cloud data,some of which is not actually part of the space. Unlike conventionalsystems that obtain a large amount of point cloud data to be processedin order to generate a 3D representation of a space, the systemdescribed herein is able to generate a 3D representation of that samespace while minimizing the amount of data needed. Furthermore, thesystem described herein is able to generate a 3D representation of aspace regardless of any amount of clutter within the space, so long aseach surface of important structural features of the space is at leastpartially exposed.

Furthermore, because the system described herein uses a user-guidedapproach to identifying bounds of the space, the system is able toaccount for structural features that conventional systems are not ableto address. For example, conventional systems are often unable todistinguish atypical structural features (e.g., beams, protrusions,depressions, etc.) from obstacles or clutter. The current system is ableto account for these atypical structural features using guidanceprovided by a user. For example, in the system described herein, a usermay indicate each surface of each structure, even atypical structures,which may then be reflected within a 3D representation generated usingthe system. Conventional systems often require that an initial 3Drepresentation generated by the system be manually altered by a user toinclude any atypical structural feature. This can result in a loss ofaccuracy, as users may not accurately generate the feature within therepresentation.

The various embodiments further can be implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers, computing devices or processing devices which can be used tooperate any of a number of applications. User or client devices caninclude any of a number of general purpose personal computers, such asdesktop or laptop computers running a standard operating system, as wellas cellular, wireless, and handheld devices running mobile software andcapable of supporting a number of networking and messaging protocols.Such a system also can include a number of workstations running any of avariety of commercially-available operating systems and other knownapplications for purposes such as development and database management.These devices also can include other electronic devices, such as dummyterminals, thin-clients, gaming systems, and other devices capable ofcommunicating via a network.

Most embodiments utilize at least one network that would be familiar tothose skilled in the art for supporting communications using any of avariety of commercially-available protocols, such as TransmissionControl Protocol/Internet Protocol (“TCP/IP”), Open SystemInterconnection (“OSI”), File Transfer Protocol (“FTP”), Universal Plugand Play (“UpnP”), Network File System (“NFS”), Common Internet FileSystem (“CIFS”), and AppleTalk. The network can be, for example, a localarea network, a wide-area network, a virtual private network, theInternet, an intranet, an extranet, a public switched telephone network,an infrared network, a wireless network, and any combination thereof.

In embodiments utilizing a Web server, the Web server can run any of avariety of server or mid-tier applications, including Hypertext TransferProtocol (“HTTP”) servers, FTP servers, Common Gateway Interface (“CGI”)servers, data servers, Java servers, and business application servers.The server(s) also may be capable of executing programs or scripts inresponse to requests from user devices, such as by executing one or moreWeb applications that may be implemented as one or more scripts orprograms written in any programming language, such as Java®, C, C#, orC++, or any scripting language, such as Perl, Python, or TCL, as well ascombinations thereof. The server(s) may also include database servers,including without limitation those commercially available from OracleMicrosoft Sybase®, and IBM®.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (“SAN”) familiar to those skilledin the art. Similarly, any necessary files for performing the functionsattributed to the computers, servers, or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (“CPU”), at least oneinput device (e.g., a mouse, keyboard, controller, touch screen, orkeypad), and at least one output device (e.g., a display device,printer, or speaker). Such a system may also include one or more storagedevices, such as disk drives, optical storage devices, and solid-statestorage devices such as random access memory (“RAM”) or read-only memory(“ROM”), as well as removable media devices, memory cards, flash cards,etc.

Such devices also can include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired)), an infrared communication device, etc.), and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium, representing remote, local, fixed, and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting, and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services, or other elementslocated within at least one working memory device, including anoperating system and application programs, such as a client applicationor Web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets), or both. Further, connection to other computing devicessuch as network input/output devices may be employed.

Storage media computer readable media for containing code, or portionsof code, can include any appropriate media known or used in the art,including storage media and communication media, such as but not limitedto volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules, or other data, including RAM, ROM, ElectricallyErasable Programmable Read-Only Memory (“EEPROM”), flash memory or othermemory technology, Compact Disc Read-Only Memory (“CD-ROM”), digitalversatile disk (DVD), or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage, or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by a system device. Based on the disclosureand teachings provided herein, a person of ordinary skill in the artwill appreciate other ways and/or methods to implement the variousembodiments.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the disclosure asset forth in the claims.

Other variations are within the spirit of the present disclosure. Thus,while the disclosed techniques are susceptible to various modificationsand alternative constructions, certain illustrated embodiments thereofare shown in the drawings and have been described above in detail. Itshould be understood, however, that there is no intention to limit thedisclosure to the specific form or forms disclosed, but on the contrary,the intention is to cover all modifications, alternative constructions,and equivalents falling within the spirit and scope of the disclosure,as defined in the appended claims.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosed embodiments (especially in thecontext of the following claims) are to be construed to cover both thesingular and the plural, unless otherwise indicated herein or clearlycontradicted by context. The terms “comprising,” “having,” “including,”and “containing” are to be construed as open-ended terms (i.e., meaning“including, but not limited to,”) unless otherwise noted. The term“connected” is to be construed as partly or wholly contained within,attached to, or joined together, even if there is something intervening.Recitation of ranges of values herein are merely intended to serve as ashorthand method of referring individually to each separate valuefalling within the range, unless otherwise indicated herein and eachseparate value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g., “such as”) provided herein, isintended merely to better illuminate embodiments of the disclosure anddoes not pose a limitation on the scope of the disclosure unlessotherwise claimed. No language in the specification should be construedas indicating any non-claimed element as essential to the practice ofthe disclosure.

Disjunctive language such as the phrase “at least one of X, Y, or Z,”unless specifically stated otherwise, is intended to be understoodwithin the context as used in general to present that an item, term,etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y,and/or Z). Thus, such disjunctive language is not generally intended to,and should not, imply that certain embodiments require at least one ofX, at least one of Y, or at least one of Z to each be present.

Preferred embodiments of this disclosure are described herein, includingthe best mode known to the inventors for carrying out the disclosure.Variations of those preferred embodiments may become apparent to thoseof ordinary skill in the art upon reading the foregoing description. Theinventors expect skilled artisans to employ such variations asappropriate and the inventors intend for the disclosure to be practicedotherwise than as specifically described herein. Accordingly, thisdisclosure includes all modifications and equivalents of the subjectmatter recited in the claims appended hereto as permitted by applicablelaw. Moreover, any combination of the above-described elements in allpossible variations thereof is encompassed by the disclosure unlessotherwise indicated herein or otherwise clearly contradicted by context.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

What is claimed is:
 1. A method of generating a three-dimensional (3D)representation of a space comprising: receiving an indication of anumber of points, each of the points corresponding to a location upon asurface of a structural feature within the space; determining, for eachof the number of points, a number of corresponding boundaries that matchthe surface of the corresponding structural feature for at least someamount of area; identifying, from the determined number of correspondingboundaries, multiple pairs of intersecting boundaries; generating a setof junctions, wherein each junction is generated as an intersection of apair of intersecting boundaries of the multiple pairs of intersectingboundaries; and after determining that each of the number ofcorresponding boundaries are completely limited in extent by junctionswithin the set of junctions, generating the 3D representation of thespace using the set of junctions.
 2. The method of claim 1, wherein eachof the points are determined to correspond to the location upon thesurface of the structural feature based on depth information received inrelation to the point.
 3. The method of claim 2, wherein the depthinformation received in relation to the point corresponds to imageinformation.
 4. The method of claim 2, wherein the depth information isobtained via a depth sensor installed upon a user device.
 5. The methodof claim 1, wherein the 3D representation is a wireframe model of thespace.
 6. The method of claim 1, wherein the 3D representation isgenerated by a computer aided drafting application using the set ofjunctions.
 7. The method of claim 1, wherein the indication of thenumber of points is received at a server from a user device.
 8. A systemcomprising: one or more camera devices; a processor; and a memoryincluding instructions that, when executed with the processor, cause thesystem to, at least: obtain, from the one or more camera devices, depthinformation associated with a scene; receive an indication of a pointwithin the depth information; calculate, using the depth information, afirst boundary associated with the point; determine one or more boundsfor the first boundary based on at least one second boundary that isobtained in relation to the scene and that intersect with the firstboundary; and generate a three-dimensional (3D) representation of thescene based at least in part on the one or more bounds.
 9. The system ofclaim 8, wherein the one or more camera devices are in a user device andwherein the processor and the memory are in a mobile application serverin communication with the user device.
 10. The system of claim 8,wherein the 3D representation of the scene includes at least arepresentation of an atypical structural feature.
 11. The system ofclaim 10, wherein the atypical structural feature is one of a beam,protrusion, or depression.
 12. The system of claim 8, wherein the firstboundary matches a surface of a structural feature for at least someportion of distance.
 13. The system of claim 12, wherein the secondboundary matches a second surface of the structural feature for at leastsome portion of distance.
 14. An apparatus comprising: a camera device;a depth sensor device; a mobile application stored in acomputer-readable medium that, when executed, causes the apparatus to,at least: receive depth information from the depth sensor device whichcorresponds to image information captured using the camera device;receive an indication, via the image information, of a first point and asecond point within the depth information; identify, using the depthinformation, a first boundary associated with the first point and asecond boundary associated with the second point within the depthinformation; determine a junction as a line on which the first boundaryand second boundary intersect; and cause a three-dimensional (3D) modelto be generated that includes at least the determined junction.
 15. Theapparatus of claim 14, wherein the apparatus further comprises a displayand the indication of the first point and the second point are receivedas a selection of corresponding points on the display.
 16. The apparatusof claim 14, wherein the 3D model is of a bounded space that theapparatus is within.
 17. The apparatus of claim 14, wherein the firstpoint and the second point are on different surfaces of one or morestructural features.
 18. The apparatus of claim 17, wherein a positionof the first point and a position of the second point are indicated inrelation to a single point of origin.
 19. The apparatus of claim 14,wherein the 3D model is caused to be generated by a mobile applicationserver.
 20. The method of claim 1, wherein the 3D representation iscaused to be generated by a mobile application server.